Field
The present application relates generally to communication networks, and more particularly to data center networks with improved interconnections and improved interconnection management.
Description of the Related Art
Communication networks have a long history, evolving from single transmission lines and manual switching, to early multi-line automatic electro-mechanical switching systems, to more recent electronic and optical transmissions across many lines or fibers using electronic or optical switching systems.
Today's digital and optical switching systems allow for substantial growth in the size of communication networks to meet the needs of ever expanding communication networks. The progression to the more common digital and optical switching systems was spurred on a belief that newer semiconductor (e.g., VLSI) and optical devices met the need for high speed data transmissions.
With the evolution of communication switching systems has been the evolution of computers and the information age. In order to manage the increase in data transmissions between computers, data centers came to be. Data centers have their roots in the huge computer rooms built during the early ages of the computing industry. Early computer systems were complex to operate and maintain, and required a special environment in which to operate. During the boom of the microcomputer industry in the 1980s, computers started to be deployed everywhere and systems, such as dedicated computers or servers, were developed to meet the demands created by the need to have the increasing number of computers communicate. During the latter part of the 20th century and early part of the 21st century, data centers grew significantly to meet the needs of the Internet Age. To maintain business continuity and grow revenue, companies needed fast Internet connectivity and nonstop operations to establish a presence on the Internet.
Today, data centers are built within the enterprise network, a service provider network, or a shared, colocation facility where the networks of many disparate owners reside. With the significant increase in business and individual use of the Internet, and the significant need for bandwidth to transmit high volumes of data, especially video and graphics, data centers are again under pressure to evolve to handle the boom in growth. However, data centers are typically very expensive to build, operate and maintain, and data center operators are searching for ways to reduce costs while increasing data processing and transmission capabilities, while meeting all reliability requirements.
In order to meet the increased demands, data center network architectures have changed. Sometimes the changes to the network architecture require significant rerouting of network connections, and sometimes the network architecture needs to be dynamic, changing frequently. And, all this has to be achieved at today's fast rates with little or no failures or delays in the transmission of data.
One area where the data center network is changing is with network switches that have evolved with the capability of switching data traffic on a packet-by-packet basis, which is known as packet switching. While packet switching can change the physical route of individual packets through a network, there are some network applications where the requirement is to switch all the data traffic from one physical route to a second physical route through the network, which is known as port switching or path switching.
Traditionally, data center network devices, such as servers, storage devices, switches, and routers, as well as NIC cards that may be added to such devices have physical connection points to transmit and receive data. These connection points generally include a transceiver and a connector, which are often referred to as a port. Ports can be copper or fiber ports that are built into the device, or the ports can be plug-in modules that contain the transceiver and connector and that plug into Small Form Factor (SFF) cages intended to accept the plug-in transceiver/connector module, such as SFP, SFP+, QSFP, CFP, CXP, and other transceiver/connector modules, where the connector extends from an exterior surface of the device, e.g., from a front panel. Fiber ports may be low density or single fiber ports, such as FC, SC, ST, LC, or the fiber ports may be higher density MPO, MXC, or other high density fiber ports.
Fiber optic cabling with the low density FC, SC, ST, or LC connectors or with SFP, SFP+, QSFP, CFP, CXP or other modules either connect directly to the data center network devices, or they pass through interconnector cross connect patch panels before getting to the data center network devices. The cross connect patch panels have equivalent low density FC, SC, ST, or LC connectors, and may aggregate individual fiber strands into high density MPO, MXC or other connectors that are primarily intended to reduce the quantity of smaller cables run to alternate panels or locations.
From a logical perspective, traditional data center networks, as shown in FIG. 1, includes of servers 104 and storage devices 106, plus connections between the servers, storage devices and to external interfaces. A data center interconnects these devices by means of a switching topology implemented by pathway controlling devices 130, such as switches and routers. As networks grow in size, so does the complexity. The servers 104 and storage devices 106 connect to one another via cable interfaces 118, 120, 122, and 124. Interconnects 112 are used to bundle and reconfigure cable connections between endpoints in cable bundles 114, 116, and 126. The Management Controller 100 configures and controls and receives status information from the data center network devices via management interface path 101. As can be seen in FIG. 1, data center networks become layered with multiple pathway controlling devices 130 in an attempt for every endpoint to have the capability of switching and/or routing data packets to any other endpoint within the data center network. This can result in very complex hierarchical switching networks which in turn require considerable power and expense in order to maintain and respond to configuration changes within the network.
From a physical perspective, a typical data center network configuration, shown in FIG. 2, includes multiple rows of cabinets, where each cabinet encloses a rack of one or more network devices, e.g., switches 102, servers 104 and storage devices 106. Typically, for each rack there is a top-of-rack (TOR) switch 102 that consolidates data packet traffic in the rack from each server 104 and storage 106 via cables 140 and transports the data packet traffic to a switch known as an end-of-row (EOR) switch 108 via cables (not shown). The EOR switch is typically larger than a TOR switch, and it processes data packets and switches or routes the data packets to a final destination or to a next stage in the data center network, which in turn may process the data packets for transmission outside the data center network. Typically, there are two TOR switches 102 for every rack in a row, e.g. Rows 1 and 2, and two EOR switches 108 for each row, where the second switch in each case is typically for redundancy purposes.
In one configuration, a TOR switch 102 will switch data packet traffic directly between any two network devices, e.g., servers 104 or storage devices 106, within a given rack. Any data packet traffic destined for locations outside of the rack is sent to the EOR switch 108. The EOR switch 108 will send data packet traffic destined for a network device in a different rack in the same row to the TOR switch 102 of the rack where the network device resides. The TOR switch 102 within the destination rack will then forward the data packet traffic to the intended network device, i.e., the destination device. If the data packet traffic is for network devices outside of the row, e.g., Row 1, the EOR switch 108 will forward the traffic to core switch 110 for further transmission.
In other configurations, a TOR switch 102 may be used as an aggregator, where all data packet traffic is collected and forwarded to an EOR switch 108. The EOR switch then determines the location of the destination network device, and routes the data packet traffic back to the same TOR switch 102 if the data packet traffic is destined for a network device in that rack, to a different TOR switch 102 in a different rack if the traffic is destined for a network device in a different rack in the same row, or to the core switch 110 if the destination of the data packet traffic is outside of that row.
The TOR switch 102 may couple the entire data packet traffic from an ingress port to an egress port, or may selectively select individual packets to send to an egress port. Referring to FIG. 3, in conventional applications, a TOR switch 102 retrieves header information of an incoming data packets on an ingress port of the TOR switch, and then performs Access Control List (ACL) functions to determine if a packet has permission to pass through the TOR switch 102. Next, a check is run to see if a connection path was previously based on the information from within the packet header. If not, then TOR switch 102 may run Open Shortest Path First (OSPF), Border Gateway Protocol (BGP), Routing Information Protocol (RIP), or other algorithms to determine if the destination port is reachable by the TOR switch 102. If the TOR switch 102 cannot create a route to the destination network device, the packet is dropped. If the destination network device is reachable, the TOR switch 102 creates a new table entry with the egress port number, corresponding egress header information, and forwards the data packet to the egress port. Using this methodology, the TOR switch 102 transfers, or switches, the data packet from the ingress port to the required egress port.
Traditional data center architectures have not had the capability to map out the physical interconnections between pathway controlling devices 130, servers 104, storage devices 106, and other devices in the data center network. Existing network applications, such as Address Resolution Protocol (ARP), Spanning Tree, OSPF and others, map out logical interconnections between two devices connected together, but such network applications do not provide information about the physical interconnections. As a result, in the event of a link failure, the end devices are aware of the failure, but cannot identify the physical interconnection which requires repair.